The Long Term Evolution based Vehicle to Everything (LTE-V2X) standard follows the LTE standard's frame format and uses a block-type pilot assisted Single-Carrier Frequency-Division Multiple Access (SC-FDMA) system for channel estimation. However, due to the time-varying characteristics of the V2X channel, large technical challenges are brought to the channel estimation at the receiver. Therefore, a time-varying channel estimation method based on sliding window filtering and polynomial fitting was designed. Aiming at the noise problem at pilot symbols, based on Least Squares (LS) method, an adaptive-length sliding window filtering was adopted for noise reduction, so as to ensure the channel estimation accuracy of pilot symbols. Furthermore, according to the size of the Doppler frequency shift of data symbols, an adaptive-order polynomial fitting method was designed to track the channel changes at data symbols. The simulation results show that the proposed method has a good denoising effect based on LS method. In the case of low-speed movement, the estimation accuracy of the proposed method is between those of LS method and Linear Minimum Mean Square Error (LMMSE) method. In the case of high-speed movement, the proposed method can fit the time-varying channel characteristics better, and its performance exceeds that of the channel estimation method of LMMSE method combined with linear interpolation. The above results show that the proposed method has better adaptability than the comparison methods and is suitable for LTE-V2X communication scenarios with different channel noises and terminal moving speeds.